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MEASUREMENT OF MESSAGE DIFFUSION FROM LEAFLETS 


Although other tests of validity and relia- 
bility might be considered more desirable 
than those employed, the exigencies of the 
situation and the nature of the data pre- 
cluded some of them. Nevertheless, on the 
basis of the preceding tests, it may be con- 
cluded that the Role Conception Inventory 
is reasonably dependable though necessarily 
crude. 


CONCLUSION 


The Role Conception Inventory provides 
the sociai psychologist with an opportunity 
to obtain essentially qualitative data from 
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large numbers of persons. The establishment 
of types based on preliminary research en- 
ables cultural accuracy in the formulation of 
the statements of the Inventory. The use of 
an inventory rather than a scale enables the 
researcher to get at contradictory or con- 
fused role conceptions which could hardly be 
ascertained were scalability his aim. The 
simple scoring technique facilitates statistical 
testing. Therefore, it seems that future em- 
ployment of the Role Conception Inventory 
may refine it into a useful tool for social 
psychological research into the nature of 
roles. 


THE COMPARATIVE VALIDITY OF TELEPHONE AND FACE- 
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behavior be depended upon to indicate 
a person’s action tendency? What is 
the degree of relationship between verbal 
and non-verbal overt behavior? These two 
inquiries pose the critical problem of validity 
in the general form that long has challenged 
investigators who use questionnaires in gaug- 
ing human behavior. 
One of the factors influencing the ability of 
a questionnaire to measure what it is de- 
signed to measure is the method of con- 
ducting the interview. A great deal has been 
written about the relative advantages of 
two of these methods—telephone and face-to- 
face interviews—particularly with respect to 
comparative costs, sampling problems, inter- 
viewer reliability, refusal rates, and ease of 
administration. Surveys conducted by tele- 
phone interviews have had wide popularity 
in recent years, especially in radio audience 
research. A special advantage of the tele- 
phone interview is the rapidity of the field 
work. Expense is likely to be considerably 
less, in terms of cost per completed inter- 
view, than in the face-to-face interview 


T: what extent can verbal or symbolic 


1See, for example, Parten, Mildred, Surveys, 
Polls, and Samples, Harper: 1950, pp. 79-93. 


methods. It has also been argued that if 
facts, rather than opinions, are desired, the 
telephone approach has particular value.? 
While considerable discussion of this sort 
is available, very little empirical data on the 
comparative validity of the telephone and 
face-to-face techniques on either opinion or 
fact issues has been recorded. This paper 
presents some data on this difficult problem 
from a survey in which two comparable sam- 
ples were interviewed with the same ques- 
tionnaire, by the same interviewers, but by 
the two different approaches. The verbal 
claims and intentions of one group of re- 
spondents interviewed face-to-face in their 
homes will be compared with the verbal 
claims and intentions of another group inter- 
viewed by telephone. These statements will 
then be matched with the actual conduct of 
both groups, permitting an estimate of the 
comparative validity of the two interviewing 
methods. 


THE SURVEY SITUATION 


The Washington Public Opinion Labora- 
tory, under an Air Force contract, is in- 


2 Blankenship, A. B., Consumer and Opinion Re- 
search, Harper: 1943, p. 48. 
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FicurE 1. THE LEAFLET 


URGENT 
IF THIS WERE AN ENEMY LEAFLET 


dropped to warn you of an atomic attack coming today, what 
would you do? 


CHECK: _....Call the Office of Civil Defense 
eee Stay home 


.._.Go to emergency center 
oo Leave city (destination)__..................-..--..2..0-.02.- 


_....Other (explain) _......-.-.- 2 ee. 


Actually this leaflet was dropped from friendly aircraft as a civil 
defense test, and your help is necessary. 


FILL IN: 1. How did you get this leaflet? 
a From another person 
ee Picked it up from the ground 


2. Where did you get this leaflet? 


ae ee © Gree we CC @ CewoeReWwere we we oo ow @ woes ew oe we & HB e  @ oo oe ose ww | & Be 


(Write in nearest street address or intersection) 
3. When did you get this leaflet? 
Day of the week... 


Time of day..._..... _--.. --. ----_ - --- + ---- 
(hour and a.m. or p.m) 


4. Male... Female____ 
PASS ON extra leaflets you find to friends whom you would 
want to warn if there were going to be an enemy attack. 


MAIL IN. It is VERY IMPORTANT that you put this leaflet in the 
nearest mailbox as soon as possible after you have completed it. 


See OW flood le 


State Security Officer 


vestigating how far, how fast, and how 
faultlessly leaflet-messages dropped from air- 
craft are likely to spread through a popula- 
tion under varying conditions.* In one large 
scale pre-test, 55,000 leaflets were released 


8 Dodd, Stuart C., “Testing Message Diffusion 
from Person to Person,” paper read at American 
Association for Public Opinion Research meeting, 
Princeton, June, 1951. 


a Govemor of Utah 


over Salt Lake City.* Half the leaflets were 
dropped over a residential area about 2 a.m., 
and the other half, identical in content except 
for a single code mark, were dropped over 
another residential section of the city twelve 
hours later around 2 p.m. 

The 5%4 by 7 inch leaflet, with black 


4“Postcards from Heaven,’ Newsweek, August 
13, 1951, pp. 70-71. 
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MEASUREMENT OF MESSAGE DIFFUSION FROM LEAFLETS 


letters on yellow paper, was in the form of 
a postcard addressed to “Civil Defense, c/o 
Postmaster, Salt Lake City, Utah.” The 
leaflet encouraged response to civil-defense 
questions, and asked the reader to check 
when, where, and how he got the leaflet, and 
to mail it in. The postage was pre-paid. 


THE MAIL RESPONSE 


A total of 3,940 leaflets were mailed back 
as requested, a great majority of them on the 
same day they were dropped. Ninety-four 
per cent of the leaflets returned had com- 
plete and usable responses. While males re- 
turned about half of the leaflets, women were 
more apt to have leaflets passed-on to them. 
At least one in four of the leaflets was in- 
volved in secondary-diffusion, that is, passed- 
on from person to person, while about three 
in four were picked up from the ground by 
the respondent. According to observers, 
children were very active in the secondary 
diffusion process. The mail returns from the 
day and night drops were markedly similar. 
With an equal number of leaflets dropped 
on both areas, 52 per cent of those returned 
came from the day mission, and 48 per cent 
from the night drop. 

In designing the leaflet, care was taken 
not to inhibit response by requiring the name 
and address of the sender. However, an in- 
direct appeal for respondents to volunteer 
their names and addresses was used. On the 
address side of the leaflet, the single word 
“From” appeared in the traditional upper 
left corner followed by three blank lines. 
Apparently this suggestion tapped well-es- 
tablished writing habits as the names and 
addresses appeared on 82 per cent of all the 
leaflets returned. By eliminating from the 
base the 255 entirely blank leaflets received, 
the identity response occured on 87 per cent 
of the leaflets mailed. As will be indicated, 
this large identity response is the basis for 
the validity check to be made on the tele- 
phone and face-to-face interviews, as it 
permitted the interviews to be matched with 
the mailed-in leaflets. 


INTERVIEWING IN THE LEAFLET-DROP AREAS 


In addition to the mailed-in leaflets, a 
formal questionnaire, administered under 
state civil-defense auspices, was used to 
measure leaflet-message diffusion. Respond- 
ents were asked fifteen short-answer ques- 
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tions to determine their exposure to the 
leaflet, what content they had perceived, and 
what instructions they had acted upon. This 
measure of community reaction to the leaflet 
operation became feasible when the news- 
papers, radio stations, press services, and 
police authorities in Salt Lake City cooper- 
ated by withholding information about the 
test for 36 hours. During this time period, 
with mass-media publicity controlled, 314 
face-to-face interviews were completed in a 
night-drop area, and 55 per cent (175) of 
the respondents claimed to have seen or 
heard about the leaflets. Also during this 
time, 347 telephone interviews were con- 
ducted with respondents in a day-drop area, 
with 46 per cent (158) reporting they had 
seen or heard about the leaflets. 

The day and night drop areas had been 
selected to be comparable targets in size 
and residential composition, excluding, for 
example, the main industrial plant sites, large 
waterways, and other open-space areas. The 
interview areas within the target areas could 
not be specified until the leaflets hit the 
ground. Pin-point accuracy for dropping 
leaflets on specified targets has not yet been 
established. When the drop-zone boundaries 
were designated by ground observers operat- 
ing from mobile units, two sample areas for 
interviewing were defined. The two interview 
areas selected consisted of about twelve 
square blocks around the center of the drops, 
each including about 400 dwelling units. 

The face-to-face interviews were con- 
ducted between the hours of six and ten 
p.m., starting about sixteen hours after the 
leaflets were released over the area. The 
telephone interviewing took place between 
nine a.m. and noon the following day, about 
nineteen hours after the leaflets were dropped 
in that area. By conducting morning and 
evening interviews, the time for potential 
exposure to the leaflets prior to interviewing 
was fairly equal in both areas. In addition, 
this arrangement permitted the training and 
use of one set of interviewers and supervisors 
—an obvious economy. 

The same experienced census interviewers 
administered identical questionnaires in both 
situations. The assignment consisted of ques- 
tioning one adult in every household in the 
interview areas. As time permitted, callbacks 
were specified in both interviews. In spite 
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of morning and evening interviews, house- 
wives dominated both samples. Seven out of 
ten of the face-to-face respondents, and eight 
out of ten of the telephone respondents were 
females. 


THE TEST FOR VALIDITY 


The principal population under study in 
this paper consists of the 145 telephone and 
the 168 face-to-face respondents who said 
they had seen the leaflet. These people were 
asked, “Did you fill out the postcard (leaf- 
let) and mail it in?” By a careful and sys- 
tematic sorting of the nearly 4,000 leaflets 
returned, a check was made to see if they 
actually did mail one in. When the address 
on the mail-return leaflet could be matched 
with the address on the telephone or face- 
to-face questionnaire, the questionnaire 
statement about mailing claims and inten- 
tions could be verified and defined as “‘valid.” 
One central assumption is involved in this 
test for validity. It is assumed that leaflets 
not identifiable by address, the thirteen per 
cent indicated earlier, came at random from 
both day and night drop areas, and would 
not, therefore, bias the established validity 
proportions. 


THE VERBAL CLAIMS AND INTENTIONS 
OF RESPONDENTS 


We have reported that an equal number of 
leaflets dropped on two residential target 
areas resulted in similar mail returns. But 
how comparable are the interview samples 
selected from these areas? As a general indi- 
cation, we have noted their like size in num- 
bers, sex composition, and the fact that 
they lived in comparable residential areas 
which were selected for interviewing because 
of their similar exposure to the leaflet as 
judged by ground observers. While these fac- 
tors are of some concern, one other specific 
comparison of the samples is of primary 
importance with respect to estimating the 
validity of the telephone and face-to-face 
interview. How do the samples compare in 
statements of claims and intentions about 
mailing in the leaflet? On this important vari- 
able, the two samples are perfectly matched. 
Proportionally, their statements of claims 
and intentions about mailing in the leaflet 
were exactly the same. In the telephone and 
face-to-face samples, 30 per cent of those 
who had seen the leaflet said they Aad al- 
ready mailed it in. In addition, 26 per cent 
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of both groups said that they had not mailed 
in a leaflet, but expressed positive zntention 
to do it. The remainder of both samples, 44 
per cent, answered the mailing question with 
an unqualified “No.” These identical claims 
and intentions are summarized in Table 1. 


TABLE 1. THE VERBAL CLAIMS AND INTENTIONS OF 


TELEPHONE AND FACE-TO-FACE RESPONDENTS 
Witu Respect TO MAILING IN A LEAFLET 


“Did you fill out 


Face-to-Face Telephone 
one of the post- 
cards (leaflets) Per Per 
and mail it in?” Number cent Number cent 
Yes 51 30 44 30 
No, but positive 
intention 43 26 38 26 
No 74 44 63 44 
Totals 168 100 145 100 


VALIDATED MAILING CLAIMS 


If the respondent said he had already 
mailed in a leaflet, had he actually done so? 
While the telephone and face-to-face sample 
expressed identical mailing claims, there was 
a vast and significant difference in their veri- 
fied mailing behavior. Eighty per cent of the 
face-to-face and only 16 per cent of the tele- 
phone mailing claims were verified. The dif- 
ference between these proportions is signifi- 
cant at the .01 level. Thus we note that in 
this survey a “Yes” response to a question 
about an act already completed had very 
low validity when asked of telephone re- 
spondents, and much greater validity when 
asked face-to-face. 


VALIDATED MAILING INTENTIONS 


Possibly some of the difference between 
telephone and face-to-face “validity” can be 
explained in terms of an activating effect of 
the face-to-face interview. The more flexible, 
personal interaction of the face-to-face inter- 
view situation, with better opportunities for 
probing and explanation of purposes, might 
have influenced respondents to do a civic 
duty and mail in the civil-defense leaflet. 
This hypothesis is suggested when the mail- 
ing behavior of those who expressed positive 
intention to mail is examined. Here, again, 
an equal proportion of both samples indi- 
cated in the interview that while they had 
not yet mailed in a leaflet they certainly 
intended to do so. Fifty-one per cent of the 
face-to-face respondents who said they would 
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mail in a leaflet actually carried out their 
intentions, while only 10 per cent of the 
telephone “intenders’” followed through. 
While the difference in the validity of inten- 
tions between the two samples is not as great 
as the difference in the validity of claimed 
mailing, both differences are significant at 
the .01 level, and indicate the greater valid- 
ity of the face-to-face interview. 

Another hint of the “activating effect” of 
the face-to-face interview is indicated by the 
fact that five face-to-face respondents who 
neither made a claim nor stated an intention 
to mail the leaflet-—they were in the ‘“‘No” 
category of Table 1—nevertheless sent one 
in. None of the comparable telephone re- 
spondents were thus activated. With respect 
to the question under study, this suggests 
that the “No” telephone response was the 
single most “valid” of all the verbal state- 
ments. There is good reason to question the 
validity of the positive claims and intentions 
of the telephone respondent, but, at least, 
when the telephone respondent said “‘No” he 
meant what he said, and apparently did not 
change his mind. 


RESPONSE TO OTHER QUESTIONS 


In view of the comparative validity of 
mailing claims and intentions, it becomes of 
interest to note the telephone and face-to- 
face response to other questions in the inter- 
view for which no direct check on validity 
was attempted. Essentially, the questionnaire 
posed four kinds of questions to those who 
claimed to have seen the leaflet: 


(1) Questions on characteristics of respond- 
ents, i.e., sex, age, and 1950 family in- 
come. 

(2) Questions about initial contact with the 
leaflet, i.e., when and where? 

(3) Questions about perception and knowl- 
edge of leaflet content, i.e., what color 
was the leaflet, who signed it, what did it 
tell you to do, how was it first distrib- 
uted? 

(4) Questions about communications behav- 
ior, i.e., mailing, telling others, passing- 
on extra leaflets, making further inquiries 
about test. 


A brief comparative summary of the re- 
sponses to these questions by the 168 face- 
to-face and the 145 telephone respondents 
who said they had seen the leaflet follows. 
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CHARACTERISTICS OF RESPONDENTS 


The last three questions asked of the re- 
spondents permit classifying the samples by 
sex, and comparing their claims about age 
and family income. 

Both interview samples were predomi- 
nantly female. However, the telephone sam- 
ple, with 86 per cent, had a significantly 
greater proportion of women than did the 
face-to-face sample which was 67 per cent 
female. 

The respondents were asked, “What was 
your age as of your last birthday?” It was 
stressed that answers need be given in 10 
year group intervals only. There were no 
refusals on this question. The two samples 
are closely comparable in the older age cate- 
gories, but the telephone sample claimed to 
have more young people. The age responses 
are summarized in Table 2. 


TARLE 2. THE PROPORTION OF TELEPHONE AND 
FACE-TO-FACE RESPONDENTS IN VARIOUS 
AGE GROUPS 


Percentages of Samples 


Face-to-Face Telephone 

Age (N=168) (N=145) 
Under 21 4 9 
21-29 14 17 
30-39 26 24 
40-49 19 17 
50-59 16 13 
60 years and over 21 20 
Total 100 100 


The last question on the schedule asked, 
“What was the approximate (money) income 
of your family for 1950?” The most impor- 
tant difference in the two samples on this 
question is the greater proportion of refusals 
in the telephone sample, a difference which 
is significant at the .05 level. The economic 
status of the two samples is indicated in 
Table 3. 


INITIAL CONTACT WITH THE LEAFLET 

The questions about initial contact with 
the leaflet elicited almost identical response 
from both samples. At least nine out of ten 
respondents in both samples said they saw 
the leaflet on the same day it was dropped, 
Thursday, and saw it first at home or in their 
immediate residential neighborhood. 
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TasLeE 3. THe ProporRTION OF TELEPHONE AND 
FACE-TO-FAcE RESPONDENTS By INCOME GROUPS 


Percentages of Samples 


Family Income Face-to-Face Telephone 

1950 (N=168) (N=145) 
Under 1,000 9 7 
1,000—2 ,999 26 23 
3,000—4,999 45 38 
5,000—6,999 12 9 
7,000 and over 3 5 
No Report (refusals) 5 18 
Total 100 100 

PERCEPTION AND KNOWLEDGE OF THE 


LEAFLET CONTENT 


As may be seen in Table 4, there are 
consistent differences between the two sam- 
ples in their response to the questions meas- 
uring their perception and knowledge of the 
leaflet content. 

In the face-to-face sample, the level of 
knowledge about the leaflet content is con- 
sistently higher than in the telephone sam- 
ple. Analyzing by “non-knowing,’ we ob- 
serve that 35 per cent of the face-to-face and 
50 per cent of the telephone respondents who 
said they had seen the leaflet, could not re- 
port even one of the three things the leaflet 
told them to do (fill-in, pass-on, mail-in). 
This difference between the two samples is 
significant at the .01 level. In the same direc- 
tion, but not significantly different, is the 
contrast between the 32 per cent of the face- 
to-face and the 41 per cent of the telephone 
sample who could not remember, or incor- 
rectly knew, who had signed the leaflet. 
(The governor and the state security officer, 
both prominent officials in Utah, had their 
names written and printed on each leaflet). 
As a further indication of relative content 
perception, 23 per cent of the face-to-face 
and 28 per cent of the telephone sample did 


TaslLe 4. THe PROPORTION OF TELEPHONE AND FACE- 
TO-FACE RESPONDENTS WHO ACTUALLY KNEW 
IMPORTANT “EARMARKS” OF THE LEAFLET 
CONTENT 


Percentages of Samples 
Knowing Content 


Leaflet 
Content Face-to-Face Telephone 
“Earmarks” (N=168) (N=145) 
Instructions to Act 65 50 
Identity of Signatures 68 59 
Leaflet Delivery 77 72 
Color of Leaflet 97 92 
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not know that the leaflet was first distributed 
by airplane, although this fact was printed 
on each leaflet. Color was most accurately 
remembered. Only three per cent of the face- 
to-face and eight per cent of the telephone 
samples could not accurately remember the 
color of the leaflet. In general, color made an 
impression and content did not, but even 
here the difference between the two samples 
is significant at the .05 level. 


COMMUNICATIONS BEHAVIOR 


The Salt Lake City leaflet operation was a 
pre-test in a series of experiments designed 
to measure community-wide message diffu- 
sion. It was hoped that the leaflet would 
promote interest in civil-defense by stimulat- 
ing certain forms of communication. Table 
5 summarizes the communication claims of 
the telephone and face-to-face samples. 
TABLE 5. THE PROPORTION OF TELEPHONE AND FACcE- 


TO-FACE RESPONDENTS CLAIMING TO HAVE EN- 
GAGED IN CERTAIN Forms OF COMMUNICATION 


Percentages of Samples 
Making Claims 


Communications Face-to-Face Telephone 
Behavior (N=168) (N=145) 
Telling Others 31 42 
Passing-on extra leaflets 17 23 
Finding out more about test 8 17 


In comparing the communications behav- 
lor, we See again a consistent direction in the 
differences between the two samples. A 
greater proportion of the telephone sample 
claimed to have engaged in each act of 
message diffusion, such as making additional 
inquiries about the test, passing-on leaflets, 
and telling others about the leaflet message. 
“Telling others” means pure verbal diffu- 
sion, apart from passing-on leaflets. While 
members of the telephone sample probably 
knew less than the face-to-face sampie about 
what to act upon (Table 4), it claimed to 
have acted more (Table 5). As knowing and 
acting were very closely related in this prob- 
lem, the inconsistent role of the telephone 
sample, along with the convincing check on 
actual mailing behavior, raises serious doubts 
about the validity of the telephone response 
in this survey. It appears that great caution 
should be exercised in interpreting results 
of telephone interviews in polling problems 
of similar simple structure. 
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